Performance Drift in a Clinical Prognostic Algorithm during the SARS-CoV-2 Pandemic (preprint)

Research Objective: Health systems use clinical predictive algorithms to allocate resources to high-risk patients. Such algorithms are trained using historical data and are later implemented in clinical settings. During this implementation period, predictive algorithms are prone to performance changes (drift) due to exogenous shocks in utilization or shifts in patient characteristics. Our objective was to examine the impact of sudden utilization shifts during the SARS-CoV-2 pandemic on the performance of an electronic health record (EHR)-based prognostic algorithm. Study Design: We studied changes in the performance of Conversation Connect, a validated machine learning algorithm that predicts 180-day mortality among outpatients with cancer receiving care at medical oncology practices within a large academic cancer center. Conversation Connect generates mortality risk predictions before each encounter using data from 159 EHR variables collected in the six months before the encounter. Since January 2019, Conversation Connect has been used as part of a behavioral intervention to prompt clinicians to consider early advance care planning conversations among patients with [≥]10% mortality risk. First, we descriptively compared encounter-level characteristics in the following periods: January 2019-February 2020 (pre-pandemic), March-May 2020 (early-pandemic), and June-December 2020 (later-pandemic). Second, we quantified changes in high-risk patient encounters using interrupted time series analyses that controlled for pre-pandemic trends and demographic, clinical, and practice covariates. Our primary metric of performance drift was false negative rate (FNR). Third, we assessed contributors to performance drift by comparing distributions of key EHR inputs across periods and predicting later pandemic utilization using pre-pandemic inputs. Population Studied: 237,336 in-person and telemedicine medical oncology encounters. Principal Findings: Age, race, average patient encounters per month, insurance type, comorbidity counts, laboratory values, and overall mortality were similar among encounters in the pre-, early-, and later-pandemic periods. Relative to the pre-pandemic period, the later-pandemic period was characterized by a 6.5-percentage-point decrease (28.2% vs. 34.7%) in high-risk encounters (p<0.001). FNR increased from 41.0% (95% CI 38.0-44.1%) in the pre-pandemic period to 57.5% (95% CI 51.9-63.0%) in the later pandemic period. Compared to the pre-pandemic period, the early and later pandemic periods had higher proportions of telemedicine encounters (0.01% pre-pandemic vs. 20.0% early-pandemic vs. 26.4% later-pandemic) and encounters with no preceding laboratory draws (17.7% pre-pandemic vs. 19.8% early-pandemic vs. 24.1% later-pandemic). In the later pandemic period, observed laboratory utilization was lower than predicted (76.0% vs 81.2%, p<0.001). In the later-pandemic period, mean 180-day mortality risk scores were lower for telemedicine encounters vs. in-person encounters (10.3% vs 11.2%, p<0.001) and encounters with no vs. any preceding laboratory draws (1.5% vs. 14.0%, p<0.001). Conclusions: During the SARS-CoV-2 pandemic period, the performance of a machine learning prognostic algorithm used to prompt advance care planning declined substantially. Increases in telemedicine and declines in laboratory utilization contributed to lower performance. Implications for Policy or Practice: This is the first study to show algorithm performance drift due to SARS-CoV-2 pandemic-related shifts in telemedicine and laboratory utilization. These mechanisms of performance drift could apply to other EHR clinical predictive algorithms. Pandemic-related decreases in care utilization may negatively impact the performance of clinical predictive algorithms and warrant assessment and possible retraining of such algorithms.

Impact of the COVID-19 pandemic on treatment patterns for US patients with metastatic solid cancer (preprint)

BackgroundThe COVID-19 pandemic has led to delays in patients seeking care for life-threatening conditions; however, its impact on treatment patterns for patients with metastatic cancer is unknown. We assessed the COVID-19 pandemics impact on time to treatment initiation (TTI) and treatment selection for patients newly diagnosed with metastatic solid cancer. MethodsWe used an electronic health record-derived longitudinal database curated via technology-enabled abstraction to identify 14,136 US patients newly diagnosed with de novo or recurrent metastatic solid cancer between January 1 and July 31 in 2019 or 2020. Patients received care at [~]280 predominantly community-based oncology practices. Controlled interrupted time series analyses assessed the impact of the COVID-19 pandemic period (April-July 2020) on TTI, defined as the number of days from metastatic diagnosis to receipt of first-line systemic therapy, and use of myelosuppressive therapy. ResultsThe adjusted probability of treatment within 30 days of diagnosis [95% confidence interval] was similar across periods: January-March 2019 41.7% [32.2%, 51.1%]; April-July 2019 42.6% [32.4%, 52.7%]; January-March 2020 44.5% [30.4%, 58.6%]; April-July 2020 46.8% [34.6%, 59.0%]; adjusted percentage-point difference-in-differences 1.4% [-2.7%, 5.5%]. Among 5,962 patients who received first-line systemic therapy, there was no association between the pandemic period and use of myelosuppressive therapy (adjusted percentage-point difference-in-differences 1.6% [-2.6%, 5.8%]). There was no meaningful effect modification by cancer type, race, or age. ConclusionsDespite known pandemic-related delays in surveillance and diagnosis, the COVID-19 pandemic did not impact time to treatment initiation or treatment selection for patients with metastatic solid cancers.